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INTRODUCTION 

Assessing growth in 5>;jdents* ability to read is a problem that 
has faced school systems for decades. Most schools have responded by 
regularly administering standardized tests of reading achievement. 
Educators and parents alike recognize the inadequacy of these erasures , 
for they Jo not address a most crucial questi^, i-e-, does m>' child" 
read ^ell enough to thrive m school and society. 

Recently a new approach to this xoeasurement problem was devel- 
oped by John Bormuth of the University of Chicago (Borrauth, 1S^71). 
Bonnuth, assuming that reading can at least be defined as gaining in- 
formation fron specific printed messages « developed a test of 
information gain that utilized a ntw kind of achievement test item. 
This measure is a completion type test item that can be systematically 
generated from a written message using a set of linguistic rules. 
(Bormuth, 1970b) Such items avoid most of the criticisms, norraally 
leveled at other kinds of test items when used in a criterion-referenced 
test. 

As this test of ''information gain*' was being developed, it became 
clear that the large developmental cost associated with the measure 
prohibited its general use as a program assessment tool by school 
systems. Therefore, Bormuth identified a much cheaper surrogate 
saeasurc in the cloze test which had many of the same psychometric 
qualities of the test items for his test of information gain. He found, 
moreover, that a cloze test and an "information gain*' test on the same 
material had a systematic and highly predictable relationship. More 
specifically, Bormuth fotmd that it was not until students* cloze 



scores reached about 35 that they were able to gain nuch if any infer- 
nation froa* what they had read. Thus, a criterion or threshold of minima 
reading literacy was suggested for specific reading materials. 

This realiiaticn presented school systems with an inexpensive 
assessment model for determining if students can read (can gain infor- 
mation from) what they are expected by society to read, h^hat follows is 
one school system's atterapt to pilot that assessment model under 
operational conditions. This paper represents but a brief abstract of 
the complexities of the ultimate* design and the reader is referred tc 
the final report for a more thorough description (Hansen and Hesse, 1974) 

DESIGN 

Question 

The question addressed was, "Can students in the Madison Public 
Schools achieve a correct replacement score greater than 35\ on cloze 
tests developed from samples of reading mater als which they (the student 
are expected by the community and school to be able to read?** 

instrumentation 

Since the cloze tests to be used m the assessment were' co be based 
or> reading material which students in our school system were actually 
e)q)ectcd to read, it was necessary to develop the testing material for 
the assessment. Briefly, the following procedure was used. With the 
help of reading specialists and language arts teachers^ a taxonomy of 
reading materials was developed, (Tables 1.1, 1.2, and^J.3) That 
taxonomy included 10 reading domains: (a) Recreational Lif^ra^ure, 
(b) School, (c) Automobile, (d) Citizenship^ (e) Leisure-time Activities, 
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(f) Occupational ^'.aterial, (g) Reference Materials, (hj Safety, 

{ij Consumer >taterial , and (j) Textbooks. Spread across these ten dootains 

were 60 categories of laateriaK (See Tables 1.1, 1.2, and 1.3 for the 

breakdown . ) 

TABLE LI 
DOKAIKS AND CATEGORIES 

A. Recreational Literature : Grade Level Used 

1. Magazines (3) 4-7-10-12 

2. Stories dealing with family relationships (3) 4-7-10-12 

3. Animal stories (3) 4-7-10-12 

4. Patriotic stories (3) 4-7-10-12 

5. Biooraphies (3) 4-7-10-12 

6. Aiventure stories (3) 4-7-10-12 

7. Student newspapers (3) 4-7-10-12 

8. Mystery stories (3) 4-7-10-12 

B. School : 

1. Student handbook (3) . 7-10-12 

2. Paperback list (15) 4-7-10-12 

3. Standardized test instructions fS) 4-7-10-12* 

C. Automobi le : 

1. Proroocional literature on ne%: cars (J) 10-12* 

2. Auto insurance promotion (3) 10-12* 

3. Automotive license manual (3) 10-12* 

4. Automotive driving tips (3) 10-12* 

5. Operator's and owner's manual (3) '10-12* 

6. Penalty point literature (3) 10-12* 

D. Citiieiiship : 

1. Wisconsin Constitution (3) 10-12* 

2. Voting directions (3) 10-12* 
3- Newspapers (40) 7-10-12* 
4. Referenda statements (27) 10-12* 



( ) Number indicates the quantity of passages which will represent that 
category. 

* Indicates all grades involved received the same passages » 



TABLE 1.2 . 
DOMAINS AND CATEGORIES 



Leisure- tipie Activities : GraJe Level Used 

1. Rule books (sports) (3) 7--.0-l2» 

2. Directions for assembling toys (3) 4-7-10-«42* 
5. Recreation department bulletins (3) 4-7-10-T2 

4. Directions for game? (3) 4-7-l6*^i2* 

5. T.V. Guide (3) 4-7-1C-12* 

6. Boy/Girl Scout Manual (3) 4-7-10-12* 

7. Directions acconpanying sewing patterns (3) 4-7-10-12* 



Occupational : 



I. 


Vocational School annual ad (3) 


10- 


12* - 


2. 


Instructions on job applications (3) 


10- 


12* 


3. 


Civil Service test instructions (3) 


10- 


12* 


4. 


Armed Forces promotional literature f3j 


10- 


12* 


5. 


Prospective careers promotional literature (3) 


10- 


12* 


6. 


Instructions for filling out job- related 








forms (3) 


10- 


12* 


7. 


School catalogs (3) 


10- 


12* 



Reference Materials: 



1. Road maps (3) 7.;0-12* 

2. Telephone directory (3) 4-7-10-12* 

3. Encyclopedia (3) 4-7-10-12 

4. Reference books (3) 4-7-10-12 

Safety: 



1* Fire Department literature (3) 4-7-10-12* 

2. Airplane einergency literature (3) 4-7-lD)-12' 

3. Civil Defense instructions (3) ' 4-7-10-12*^ 

4. Bicycle and pedestrian rules and fire 

evacuation (5) 4-7-10-12* 

5. Warnings and instructions on commercial 

packaging (10) 4-7-10-lU* 

6* Heart, Cancer, and Red Cross literature (3) 7-10- 12* 

7i Directions for power tools (3) 7-10-12* 

Consuiner Material: 



1. Junk mail (3) 7-10-12* 

2. Messages on packages (3J . 4-7-10-12* 

3. Recipes (3) 7-10-12* 

4. Catalogs (3) 7-10-12* 

5. Cwitractual agreements (3) lC-12* 

6. Banking proirxjtional literature (3) 10-12* 

7. Financial planning literature (3) 10-12* 
8* Advertisements (3) 7-10-12* 
9* Conservation/ecology literature (3) 7-10-12* 

10. Consumer magazines (3)* . 10-12* 



TABLE 1.3 
DO^IAINS AND CATEGORIES 



J, Te xtbooks : Grade Level Used 

1. Language Arts and Reading (12) 4-7-10-12 

2. Social Studies (12) 4-"-10-12 
5. Science [\2) 4-7-1C-12 
4. Mathematics (12) 4-7-10-12 

Within each category of each reading domain a set of documented 
decision rules was used to obtain representative messages for each category^. 
After these roessages^ad been gathered and appropriately coded, a random 70 
word passage was extracted from each message. Since analysis of the data 
was not to be extended to the message level, it was not necessary tc 
sample multiple passages for each message. All of the passages for a 
particular reading domain made up a reading subtest for that domain. Sub- 
tests across each domain made up the tota? test« Each 70 word passage was 
clozed by eliminating every fifth word and substituting a uniformly sized 
line until a total of 10 cloze test items was identified for each passage. 

Population and Sample 

Students in grades 4, 7, 10, and 12 were selected for the assess- 
ment. A random 50* sarrple of the fourth graders was tested; all seventh, 
tenth, and twelfth graders were tested. 

D^)s ign 

It is ovidcnt from examining the categories and domains in Tables 
1.1, 1.2, and 1.5 that society may not have the S3ine expectations of student 
at all four grade levels. Thus each category within each doa^ain was 



carefully screened to determine the l<7west of th^ four grade levels at which 
students were expected to read the i&ateriaK This was accoApl^ished by 
:»ub«itting the categories within each domain to a p«inel of judges anade up 
of parents » teachers, and administrators. Moreover, it had become quite 
apparent when jnessages were being collected that for soroe categories and 
domains^ e.g., textbooks or leisure reading, the difficulty of reading 
material was controlled for each of the four ^rade levels. This was 
accounted for when messages were sajapled. Thus, not all domains and cate* 
gories were tested with all four grade levels, and soror categories and 
domains contained different reading material at each grade level. 

Tests for each grade level were constructed by identifying the 
passages within each doiaain ard category that were to be administered. These 

c 

passages were in turn randomly assigned to a series of test forms (the exact 
nunfcer of forms varied with the grade level). A multi-matrix sampling design 
was then employed to randomly assign one of these test forms to each student; 
thus each student did not respond to all passages in the total test, but 
rather to a random subsample^oT all possible passages. Table 2.0 displays 
the number of passages, test forms, and students (on the average) responding 
to each test form. 
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TABLE 2.0 

MULTI-KWrRlX SAMPLING PLAN FOP EACH GRADE LEVF.L 



Grade Lc-el 


4 


7 


10 ' 




Total Passages 


144 


216 


300 


\ 300 








\ 














Students Pc^- Passage 


60 


60, 


60 ,- 


60 


Pwsagcs Per Booklet Form 




, y 


11 


11 


(Including Common Passage) 










Number of Booklet Foruis^r 


18 


36 




30 


Overall Sample Needed 


1080 


216C 


■ i«oo \ 


1800 













A coBiDOn passage was also included in each test booklet. This passage was 
used to test the validity of the Bailti*matrix saispling design. 



Data Collection and Preparation 

Tests were administered by a cadre of trained test administrators ♦ 
AH testing occurred in a single, untimed setting. Students were allowed 
to get help from the test administrator in spelling words if they could 
verbally supply the word they wished spelled. 

Tests were scored by dfitermining the percentages of cloze items for 
which the eliminated word had been successfully replaced by each student. 
Only exact words rather than, synonyms or grairanatically similar words were 
scored correct because that was consistent with Bormuth's assessment mode] 



Results 

The original analysis design called for aggregating by domain the 
^•**ults obtained for passages tested on different subsaraples of students at 
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each grade level. However » this is a defensible procedure oni^ if the 
subsanples of students draim randOBly fron the total population are 
statistically equivalent. This hypothesis was tested using as a dependent 
measure a ten item cloze passage connonly adninistered to all students. 
A One-way Analysis of Variance was conducted among the subsamples at each 
grade level. The F-values for these tests are presented in Table 3.0. 



TABLE 3.0 

F-RATIOS FOR ONE-WAY ANCVA TESTS CWDUCTED ON A COmXi 
PASSAGE SCORE AMONG SUBSAMPLES OF STUDENTS AT EACH GRADE LEVEL 



Grade 


DF 


F-Ratio 


Grade 4 


17, 


lAl 


Grade 7 


3S, oO 


1.25 


Grade 10 


29. OO 


.73 


Grade 12 


29, 


1.29 



None of these values is significant at the .10 level; therefore subsanples 
of students at each grade have a high probability of being statistically 
significant. Thus, it was possible to aggregate by domain results ob- 
tained on subsamples of students. These data are presented in the paragraphs 
that follow. 

Tables 4,i through 4*4 display data for the four grade levels. 

Percentages of students who scored above and below the minimal literacy 

criterion of 35** are portrayed for each reading domain tested at that giyen 

grnde level. 
« 
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PERCtNTACES OF STLDF-NTS SCORING ABOVE AND BtLOh' 
T>lt 55*. CRHIRION UEVbL FOR EACH RP.ADING DOMAIN SURTEST 



TABLE 4.1 


• GRADE 4 






t 


Doma i n 


Below 35^ 


55 \ 


and 


Above 


Recreational Reading 


66% 








Schcx)l-Kelated 


48 




S2 




Lei sure- tixae Activities 


82 




18 




Reference 


62 




38 




Safety 


84 




16 




uonsumer 


/ / 








Textbook 


6S 




35 




TABLE 4.2 


> 

- GRADE 7 






• 


^ _ . — . 










Domain 


B*ilow. 35^ 


35 \ 


and 


Above 


Recreational Reading 






38% 




Scnool-Re iated 


44 




56 




Citizenship 


73 




27 




Leisure- time Activities 


54 




Jt6 




Reference 


6*^ 




36 




Safety 


48 




52 




Consu^-er 


60 




40 




Textbooks 


47 , 




S3 





r. 

- -10 

TABLE 4.3 - GRADE 10 



Donain 


Beloir 3S% 


35% and Above 


Rtfcrefitional Reading 


34% 


66% 


School-Related 


37 


63 


Autoscbile 


50 


SO 


Citizenship 


S4 


46 


Leisure -time Activities 


42 


58 


Occi^ational 


45 


55 


Reference 


57 


43 - 


oaic vy 




/ / 


Consumer 


42 


58 


1 excDooic^ 


XI 




TABLE 


c> 

4.4 - GRADE 12 




Dooain 


Below IS% 


35% and Above 




Recreational Reading 


284 


72% 


School-Related 


32 ■ - ^ 


68 


Automobile ' 


48 


52' 


Citizenship 


44 


56 


Le 1 sure-Hrirme--ActrTvi ties 


34 


66 


Occupational 


33 


67 


Reference 


46 


54 








Consumer 


32 


68 


Textbooks 


26 


74 
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Data in Tabic 4.1 suggest that large numbers of fourth grade students 
were unable to correctly fill in as aany as 35\ of ^he .cloze blanks irre- 
spcctivc of the domain of reading material on which they were tested. Even 
in the domain labeled School^Related , where fourth graders had the highest 
scores, 48'» of ihv students had cloze scores below 35%. Results for 
domains such as Leisure^Time Activities , Safety , and Consumer t where percen- 
tages of students below the 35\ criterion were 82%, 84%, and 77% 
respectively, suggest that in general fourth graders are unable to gain 
information from these materials. In addition to School-Related , fourth 
graders scored the highest on materials such as Recreational Reading , 
Reference , and Textbooks where some attempt has been made by teachers to 
match the level of difficulty of the reading material with the general 
ability of fourth graders. 

Data found in Table 4.2 suggest that at the seventh grade level there 
are, as at the fourth grade level, substantial numbers of students, irre- 
spective of the reading material domain, who are unable to fill in as many 
as 35% of the blanks in the test passages. The percentages run from a 
low of 44% foi School-Related materials to a high of 73% for Citizenship 
materials {primarily newspapers at the seventh grade level), Nptably, too, 
the number ^of seventh graders who could not complete. 55% or more of the 
b^anks of test passages from their textbooks (47%) is somewhat lower than * 
was true at the fourth grade level. Whereas two out ox ^very three fourth 
graders couid score no higher than 34% correct in test passages from text- 
books assigned to them, one out of every two seventh graders could score 
above 35 « on test passages from textbooks assigned to them.. 
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Again, it should be noted that, two of three domains where seventh 
grade students perforaed the best are related to the process of schooling 
( School-Related and T extbooks) where some efforts arc made to select 
materials which the average seventh grader can supposedly read. However, 
in spite of this screening, better than 40% of the seventh graders are 
still unable to gain information from these materials. 

The data in Tables 4.3 and 4.4 suggest that at grades 10 and 12 the 
^§cores divide more clearly between tedders and non-readers. Irrespective 
of the domain cxartined, at least 56% of the hi^ school population are 
scoring 35% or more correct^ thus suggesting that they can gain at least 
some information from the wide range of material represented in the study. 
However, disturbingly large percentages of 10th and 12th graders are 
still scoring below 35% correct and thus must be considered illiterate 
with the material in question. ^ 

An analysis of differences inong the various domains of material 
suggest thavt relative material difficulty is a larger variable in recounting 
for differences in level of performance at grades 10 and 12. At both 
grades, for example, students appear to have somewhat better scores on 
Recreational Reading , School -Related , Safety , and Textbooks than they have 
for categories relating to Automobile , Citizenship , Leisrre-Time Activities » 
Occupational , Reference ^ and Consumer . Materials in the domains of 
Citizenship and Reference materials appeared to be particularly difficult 
to 10th graders, since 54% and 57% of the students in those respective 
domains had scores below 35%. Twelfth graders also had considerable dif* 
ficulty with those two domains with 44% and 46% scoring bclcw the 35% 
criterion of literacy. 



f 
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Finally, since both 10th and 12th graders responded to tests of the 
saB>c passages of material, it is possible to draw direct comparisons 
between the: performance of the two groins on each of the douains. Except 
fcr the domain entitled AutcnK>bile , anywhere from 5 to \2\ firwer 12th 
graders had scores below 35% than did 10th graders. For Automobiles per- 
formance is virtually unchanged (SOS vs. 48%). The largest reductions 
in percentages of students below 35% occurred in those domains that proved 
particularly difficult for 10th graders: Citizenship , Occupational , 
Leisure-Time Activitie s, Reference , and Consumer . 

Discussion 

The Bonsuth model for assessing reading literacy postulates explicit 
standards for judging whether or not students can read well enough to 
wet the expectations of society, including tfie ^chools. If these stan- 
dards are accepted as reasonable, several genera) observations can be made 
and discussed from the data reported above. . 

First, it appears evident that substantial numbers of students 
(sooiewhere between 25 and 30%) leave high school unable to gain c^iCu ix 
any information from the basic material society is expecting them to read. 
Only in the case of safety materials was the percentage of 12th' grade 
students with scores below 35% as low as 17%. 

One's immediate reaction to such a conclusion is disbelief. Surely 
the repercussions of having a society where virtually 25% or more of the 
people have difficulty gaining information by reading would be greater than 
it is. That may well be true. On the other hand, the fact that the 
repercussions are no greater than they arc^ can perhaps be attributed to 
another hypotheses: most poorly reading adults are able to make 
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reasonable compensations for th«ir handicap in this itedia rich society. 
Reading may not* in fact, be as econookically important a skill as some have 
clained it is. Perhaps what is needed Is an inventory of adult reading 
habits and needs to determine how literate people have to be in order to 
prosper in society. 

A second observation that can readily be made from the data is that 
children's rcc\ding power develops ouch more sloviy than most educators and 
lay people would predict. Virtually 68% of the 4th graders and 56\ of 
the 7th graders had scores below the 35% criterion level irrespective of 
^.he domain. Even at grade IG as many as 30 to 50% of the students had 
scores below the 35% criterion threshold of literacy, although at both 
grades 10 and 12 these percentages were much more dependent upon the 
domain of material in question. 

Even on categories and domains where the material, although similar 
in content, is written to the expectations of the grade level, the growth 
in reading power is not noticeable. For example, on Recreational Reading 
66% of the 4th graders, 62% of the 7th graders, 34% of the iOth graders, 
and 28% of the 12th Graders were below 35%. The perceptions of those 
authors who write to audiences at each of the four grade levels would appear 
. to be unrealistic; students would not appear to have the level of reading 
power expected by these authors. 

A third inescapable observation is that considerable numbers of 4th j 
7th, 10th, and 12th grade students would appear to be unable to gain much 
information from the textbooks and reference materials pui'chased for their 
use in school, although in the case of textbooks and other school*related 
materials there is evidence of somewhat better control of readability than 
is true of non-school related materials. 
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If this is indeed true^ it represents a major problem for educators. 
Teachers have long sensed the probleos of having to work with materials that 
far too many of their student;s find exceedingly difficult to read. Ntorc- 
over, they have tendedf to seek ways to compensate for the problem as, for 
exas^le^ by lecturing on the material to be read, by introducing vocabulary 
by having the material read aloud in class « by giving questions in advance, 
or by using additional aids such as films and i^llmst rips to supplement the' 
reading assignment. 

The problem also presents dilemmas for educators who have the respon 
Sibil ity i>f purchasing instructional materials and for textbook companies 
who have a pecuniary interest in selling their materials. Certainly 
teachers and administ^-ators must be troubled by' e^qpenditures for printed 
instructional materials when si^>stantial numbers of students can apparently 
gain little or no information from those materials. the other hand, 
there may be no immedS'ate way out of the dilemma, since more readable 
materials which maintain grade level interest patterns are generally not 
available from textbook companies who at best have taken a rather 
lackadaisical stance toward textbook materials readability. 

Summary 

a 

t 

T1)e data in this report suggest that substantial numbers of students 
may be leaving schools unable to gain information from what society may 
expect them to read. Such data are not inconsistent with what others have 
found when using the same technique. Bormuth (1970a), for example, found 
evidence 'V'lhowing that 30 to 60 percent of the students in our better 
suburban schools read so poorly that they are able to learn little or none 
of the infonnation contained in their textbooks.*' Gundlach (1973) obtained 
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similar results In a literacy assesstaent conducted in several small 
coaommities in Wisconsin. Informally, many teachers, but particularly high 
school teachers, have with gr.eat candor assured these writers that the 
results are not njnrealistic, but do in fact reflect the reading problems 
which far too many high school stiidents have. 



standardized reading tests suggest that Madison Public School students 
generally perform fbove national norms. If these national norming samples 
are at all representative of childien across the country, then it would 
seem possi^i^ with some caution to (uggest that other communities will get 



similar literacy results. Perhaps these results dramatize the severity 
of a national reading problem that many citizens are only beginning^to 
become aware of. 



Finally, this situation i» not unique to Madison. Results from 
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